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aimed at increasing educational opportunities of homeless students in the 
United States. Only a few of these programs have been locally evaluated, and 
the evaluation process is particularly challenging for local evaluators to 
undertake. At a state and national level, commitment to evaluation for 
accountability and continuous improvement has not been strong, due to 
politicization of funding for programs. At the local level, programs rarely 
know how long students will stay in their program, and they must provide a 
diversity of services to student of all ages with highly individual needs. 
This paper considers strategies used by one local homeless education program 
in Nashville (Tennessee) to address these problems in evaluation process and 
design. A well -designed evaluation plan was made a component of program 
planning for the new grant cycle. Program participants were asked to define 
distant objectives (those for a longer period of time) and nearby objectives 
(those that could be met with one or two sessions with a student) , and then 
to define strategies for reaching these objectives. Strategies were developed 
to face the particular obstacles this program faced, obstacles that face most 
local evaluations of programs. These were: (1) a way to determine the length 

of stay of subjects and the size of the sample; (2) attempts to improve the 
accuracy of school records; (3) enhancing the diversity and availability, in 
terms of location, of services; and (4) better measurement of social 
development outcomes. State support for evaluation efforts and increased 
funding and direction at both state and national levels would help the tasks 
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Evaluation Design for Homeless Education Programs: 

A Meta-Evaluation of McKinney Programs in Tennessee 

Abstract 

Research on homelessness has long been fraught with political and methodological 
challenges (Johnson, 1989; Shlay & Rossi, 1992). At the same time, a number of 
interventions that merit evaluation in the past seven years have been funded that are 
aimed at increasing educational opportunities of homeless students in the United States. 
Only a few of these programs have been locally evaluated, and the evaluation process is 
particularly challenging for practitioners to undertake. At a state and national level, 
commitment to evaluation for accountability and continuous improvement has not been 
strong, due to politicization of funding for programs. Locally, programs rarely know 
how long students will stay in their program, and they must provide a diversity of 
services to students of all ages with highly individual needs. This paper considers 
strategies used by one local homeless education program to address these problems in 
evaluation process and design. 
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Evaluation Design for Homeless Education Programs: 

A Meta-Evaluation of McKinney Programs in Tennessee 

Introduction 

Research on homelessness has long been fraught with political and 
methodological challenges (Johnson, 1989; Shlay & Rossi, 1992). For instance, 
researchers have disagreed on definitions of homelessness. Federal law counts people 
who are “doubled-up” in public housing as homeless. Those people, however, are 
difficult for researchers to identify, as public housing officials would be obliged to evict 
families sharing their apartments with other families if they had knowledge of doubling- 
up. Homeless people living on the street, in abandoned buildings, and in cars are also 
difficult for census takers and others to identify (Dennis, 1991). Advocates have 
estimated that in some censuses of homeless people, up to 70% of people living in the 
streets were missed (Cousineau & Ward, 1990). 

Estimating the number of homeless children and youth poses its own problems 
(Burt, 1991). While recent studies show that the majority do attend public schools 
regularly (Anderson, Janger, & Panton, 1995), it is unclear across all contexts to whom 
the attendance data refer: Do they include students living in cars or in public housing? 
Do they include families living under bridges or on the streets? In some cases, although 
there is extensive demographic information available about homeless children and youth 
(e.g., Children’s Defense Fund, 1988), there is still much to be learned about the 
academic and psychosocial functioning of homeless children and youth. While Molnar, 
Rath, and Klein (1990) noted over seven years ago that it is time to move past 
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preliminary studies that describe only the scope and dimension of homelessness, much 
work remains to be done in this area. To that end, researchers need to focus their efforts 
on basic psychological research on homeless children and their families and on evaluative 
research designed to measure the effectiveness of programs and interventions that have 
been developed for homeless children and their families (Davey, 1995). 

What Is Known About Homeless Students and Programs Designed To Serve Them 

Social-Emotional Characteristics of Homeless Students 

Several studies have examined the social and emotional characteristics of 
homeless students and the special needs generated by those characteristics. Bassuk and 
Rubin (1987), for example, found that homeless students experienced high levels of 
anxiety and depression (see also Hughes & Borad, 1983). The same study also found that 
homeless students have difficulty making friends (Bassuk & Rubin, 1987). Other studies 
show that teachers report homeless students are more listless and apathetic (Ely, 1 987) 
and have difficulty with social functioning in the classroom (Timberlake, 1993). 

Other studies have focused on homeless students' lack of opportunities to develop 
healthy self-concepts and a sense of belonging (Rivlin, 1990; Timberlake & Sabatino, 
1994). In some cases, school staff may contribute to the problem by identifying 
students as homeless and tease them about their living environment, the clothes they 
wear, and poor performance (Children's Defense Fund, 1988). Homeless children and 
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youth, it seems, have difficulty developing stable friendships and a sense of themselves 
as able to act to make a difference in their environments. 

The spatial arrangements of shelters contribute to this problem of self- 
development. Both studies by Berezin (1988) and Gewirtzman and Fodor (1987) for 
example found that shelters typically do not provide play space for children, giving them 
few opportunities to develop informal peer relationships. The result of being placed in a 
shelter, where there is little privacy and where one has little that one can call one's "own," 
is a sense of loss of personal space and placelessness (Rivlin, 1 990). In addition, the 
loudness of many shelters, coupled with the inability to escape the noise, contributes to 
high anxiety and stress (Rivlin & Wolfe, 1985). According to Rivlin (1990), 

...it is clear that there is little opportunity to escape the stimulation or to 
control the space. Living under these conditions, often in neighborhoods 
that are, at the very least, unfamiliar and often dangerous, there is little 
spatial privacy. (Rivlin, 1990, p. 47) 

In shelters, finding one's own space becomes a source of conflict, and residents often 
complain about theft from other residents and the inability to trust those with whom they 
are staying (Rivlin, 1990). 

Educational Characteristics of Homeless Students 



Homeless students are typically young, ranging from pre-school through fifth or 
sixth grade. Among school-aged children, the average age is 9, and of these children, 
many are underperforming relative to their peers. Research on school performance of 
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homeless students show that these youth are significantly less likely to succeed than their 



peers with more stable housing. In a study completed with students in St. Louis, 
homeless students have been shown to score lower on standardized tests measuring the 
ability to use and produce language (Whitman, 1987). Other studies have shown that 
only a few homeless students score at grade level in math and reading (Ziesemer & 
Marcoux, 1992). They are more likely to repeat a grade (Shaver & Caton, 1984) and 
twice as likely as their housed peers to be failing in their current grade (Bassuk, Rubin, & 
Lauriat, 1986; Bassuk & Rubin, 1987). They are, moreover, four times more likely to 
drop out of school than their housed peers (Rafferty, 1991; Homes for the Homeless, 
1995). 

Whitman, Accardo, Boyert, and Kendagor (1990) suggest that homeless children 
may be less likely to be able to participate in school activities in ways that might predict 
success. They argue, 

These children's capacity to sit and attend, to process and recall, and to 
answer test questions may be further impaired by the emotional and 
transitional nature of the crisis situation. (Whitman et al., 1990, p. 516) 
Indeed, they found that homeless students were disproportionately represented at the low 
end of the Slosson Intelligence Test-Revised (SIT) (Jensen & Armstrong, 1985) as 
compared against published norms. 

Other studies of educational achievement, which compared homeless with other 
poor and mobile students, yielded more mixed results. For example, Rescorla, Parker, 
and Stolley (1991) found that homeless school-age children did not differ strongly in 
academic achievement from a comparison group of poor students, but that pre-school 
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aged children showed slower cognitive development than a comparison group and were 
less likely to be enrolled in a pre-school. The Bassuk and Rubin (1987) study described 
above, which compared poor housed students with homeless found only trends, and no 
significant differences, in rates of being retained or failing in school. 

Studies of barriers to access to schooling for homeless students yield differing 
(but overlapping results). In one study, directors of shelters and school personnel were 
asked to identify the three top barriers to access to education for homeless children 
(Dupper & Halter, 1994). They were: lack of parental involvement; inadequate 
transportation; and inadequate clothing. One of the ways parental involvement serves as 
a barrier to homeless students is that parents are preoccupied with daily survival needs; 
unwillingness to register children in school due to being unsure of duration in the shelter; 
and yielding responsibility to shelter staff. 

In another study completed by the National Law Center on Homelessness and 
Poverty (1995), homeless service providers in 20 states were asked about barriers to 
access to schooling for homeless students. Primary among barriers to school participation 
they identified are mobility, which involve frequent changes in school and residence, and 
preoccupation of meeting such basic needs as food and shelter (National Law Center, 
1995; see also Rafferty, 1991). Within school barriers identified include difficulty in 
transferring school records or obtaining birth certificates and lack of transportation, 
which forces children to change schools. Nearly 40% of respondents in the national 
study reported that delays in the transfer of school records and transportation were 
barriers to educational participation and success for homeless students (National Law 
Center, 1995). 




V 
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Other studies focused on different aspects of the social and institutional ecology 
that might contribute to homeless children and youth not attending school. Some have 
suggested that responsibility for attendance in school is often not clear among school 
officials and shelter providers (Dupper & Halter, 1994). Others have suggested that 
discriminatory remarks by teachers contribute to homeless students' problems with school 
(Nann, 1982). Others have argued that shelters provide little space for students to do 
their homework quietly and successfully (Gewirtzman & Fodor, 1987; Moroz & Segal, 
1990; Whitman et al., 1990). Finally, Moreno (1984) found that from the children's 
viewpoint, not having appropriate clothing was a perceived barrier to school attendance. 

Two of these barriers, transportation and mobility, are interrelated, and deserve 
more detailed consideration. Mobility is one of the greatest barriers to educational 
achievement (Whitman et al., 1990). A study of students in New York found that 80% of 
homeless students had changed schools at least twice in the last year (Nunez, 1994b). Of 
that 80%, 27% had attended three different schools, and 13% had attended at least four 
different schools (Nunez, 1994b). Homeless students' mobility, moreover, is not a 
temporary phenomenon, but appears to be a consistent feature of homeless students' lives. 
A report from Homes for the Homeless (1996) reveals that only 16% of homeless 
families became homeless due to one-time housing emergencies. Prior to becoming 
homeless, 80% moved two or more times within a year, and 63% lived in doubled-up 
situations with friends or relatives (Homes for the Homeless, 1996). 
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Evaluations of Interventions Aimed at Promoting Social and Academic Development 



Only a few assessments of successful shelter- or school-based social and 
educational programs for homeless students provide detail on how practical these 
suggestions are. For example, the summative evaluation of the B.F. Day School's 
program, which employs a model of interprofessional case management, focused mainly 
on demographics of participants and less on social or educational outcomes (James, 
Smith, & Mann, 1991; Washington State Coalition for the Homeless, 1989). One study, 
though not an evaluation of a specific intervention, did assess how well shelter providers 
and school staff exhibited inter-agency cooperation and coordination (Dupper & Halter, 
1994). The responses were wide-ranging, but those who rated coordination as excellent 
or very good reported examples in which there was: 

(1) direct communication between a designated shelter worker and a 
designated member of the school staff (school staff included school 
principals, guidance counselors, and school social workers); 

(2) an education advocate who works closely with both shelter and school 
staff to provide better access to school services; and 

(3) a meeting between shelter and school staff at the beginning of the 
school year to establish mutually agreed upon policies and procedures. 
(Dupper & Halter, 1 994, p. 42) 

Dupper and Halter's (1994) study also identified barriers to successful inter- 
agency coordination. When there was no designated person to ensure regular attendance 
and no procedure to track students, coordination of services for homeless students was 
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less effective. Also, when schools required birth certificates and other documents, such 
as past school records, these requirements acted as barriers to student enrollment and 
attendance. Finally, insensitivity of both school officials and shelter staff contributed to 
failure to coordinate effectively services to homeless students. 

The New York City organization Homes for the Homeless’ family-based 
approach to encouraging educational achievement among homeless students has shown 
positive educational results (Nunez, 1994b). The programs there use Residential 
Educational Training Centers, on-site supplementary schooling at shelters for homeless 
families. The centers include comprehensive job training programs for parents and for 
children and learning in an accelerated school called the Rrownstone School. An overall 
educational plan is developed for each member of the family. Family counselors serve as 
liaisons to the schools, working closely with teachers and other school personnel and 
addressing specific educational needs of students. 

The Accelerated Learning School, based on models of education developed by 
Levin (1986), stresses concept learning, analysis, and problem-solving. It also practices 
students in discourses of writing, science, and mathematics. There are several projects 
for project-based learning, such as a garden (used for teaching science) and journaling 
(used for teaching writing). The students make learning contracts which are revised 
every two weeks (Nunez, 1994b). The Brownstone School also monitors attendance of 
students (see also Social Work intervention recommendations). 

One of the strengths of the program there is the early childhood education 
program, Jump-Start, which gives pre-schoolers a chance to participate in activities 
designed to develop children's cognitive and social abilities. Homes for the Homeless 
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found that only 21% of homeless kindergarten students had ever been enrolled in a 
preschool program, compared with 60% of families with incomes of $50,000 or above. 
Homes for the Homeless' Jump-Start program showed educational gains after as little as 
eight weeks in the areas of language skills, attention span, and cooperation (Nunez, 
1994b). The Brownstone School shows similar success, showing gains in reading and 
mathematics as well as higher attendance rates (Nunez, 1994b). Adults, too, were 
inspired by their children's success: 60% earned their GED while at HFH (Nunez, 
1994b). 

The Current Research 

This paper reports the results of a meta-evaluation of two researchers’ 
collaborative work with coordinators of homeless education hired by local educational 
agencies (LEAs) in the state of Tennessee. Meta-evaluation, a method for examining or 
evaluating the process of evaluation, has the potential to ensure greater participation and 
ownership of the evaluation process by giving stakeholders a chance to comment and 
review on the evaluation process (Kemmis, 1986; Aspinwall, et al., 1992). Our meta- 
evaluation is more limited in scope than most, focused primarily on two kinds of issues 
related to the design of program evaluations for homeless education programs: (1) 
problems in the usefulness of data about homeless students available to practitioners; and 
(2) problems in the design of tools for practitioners to use to evaluate program 
effectiveness. In the last section, we examine strategies we developed collaboratively 
with the homeless education program coordinator in a large district in the state for use 
with the current year’s evaluation. 
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In our examination of issues in the design of evaluation of homeless education 
programs, we take an evaluation approach that is participatory (Cousins & Earl, 1992), 
utilization-focused (Patton, 1986; 1994), and integrated with processes of continuous 
improvement and program planning (Fetterman, Kaftarian, & Wandersman, 1996). Our 
approach is based on the idea that for program coordinators, participation in the 
evaluation process is key to ensuring that program planners and managers use evaluation 
data to support decision-making. The involvement of program managers of homeless 
education programs has the potential to encourage program staff to think more 
systematically about the relationship between program activities and objectives. Such 
systematic reflection would be aimed at building a "culture of learning” (Patton, 1997, p. 
147) to lead to continuous program improvement. 

To that end, we developed a survey of evaluation practices that was distributed to 
eleven homeless education coordinators in Tennessee in the spring, 1997. The surveys 
asked coordinators to describe the basic aims and activities of their programs, name 
current data collection activities that were a part of their program, and identify outcomes 
that coordinators wish they could measure but did not currently know how to measure. 
The surveys were used to design a training for program coordinators in program 
evaluation. During the training, coordinators were asked: 

• To identify both academic and non-academic outcomes for their program; 

• To develop a program “logic model” to indicate the relationships between 
their activities and their program objectives; 

• To choose from a list of tools (provided by us the researchers) to measure 
social development outcomes for their programs; and 
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• To develop an evaluation plan to be implemented in the 1997-98 school year. 
We used the results of the survey and the responses to the training session to generate a 
list of shared evaluation problems faced by homeless education coordinators, which are 
described in detail in this paper. 

While each district is required every year to submit an annual evaluation report, 
for the past five years only two districts have consistently done so: Memphis and 
Nashville. We include data from both districts’ reports in this meta-evaluation, though 

we believe these reports — as well as the fact that few districts have been able to complete 

1 

an evaluation of their programs — are indicative of how challenging conducting an 
evaluation of a homeless education program can be. 

McKinney Act Programs: Educational Programs for Homeless Children and Youth 

In 1987, Title VII of the Stuart B. McKinney Act authorized the creation of 
educational programs designed to serve homeless children and youth. Funding, however, 
for these programs was limited until 1 990, when Congress authorized funding to states to 
give to local educational agencies to help implement the law, which required school 
districts to provide equal educational opportunities for homeless students. While the 
program has been found to be successful in reducing barriers to school for homeless 
children and youth (Anderson et al., 1995), support has been waning in the past five 
years, and in FY95, Republicans in Congress proposed ending funding for the program 
altogether. While funding was saved, there have been significant reductions in overall 
appropriations for Title VII programs in the past three years. 




14 



EVALUATION OF HOMELESS EDUCATION 

14 

Reduced funding levels have meant that many states have scaled back the number 
of programs they can fund. The Coalition for the Homeless (1998) estimates that some 
41 local homeless education programs have been or will be eliminated: these programs 
have served some 12,800 homeless children and youth. In Tennessee, funding to 
programs fell 21% between 1995-96 and 1996-97. At the same time, as the table below 
shows, more students were served despite the reduced funding. 

Table 1. Tennessee McKinney Act Programs 

Year , 

1995-96 1996-97 

Total Funding for State $5 1 2.243 $40 1 ,9 1 7 

Number of LEAs with 11 12 

Programs 

Total Students, Served 1 5,500 5,900 

In the current year, the funding for McKinney programs in Tennessee has been changed 
to three-year cycle. Each year, however, programs will be expected to demonstrate that 
the students in their programs are accountable to the same educational outcomes all 
students in Tennessee are expected to meet. Among these expected outcomes include a 
year’s growth in achievement test scores as measured by the CTBS and high attendance 
rates (94-97%). With this increased level of accountability, however, there has not been 
an increase in support available to programs from the state level: the coordinators’ 
position has been reduced from a full-time position three years ago to a position that 
accounts for 5% of a state Department of Education staff member. The imperative to 
develop and share strategies for evaluation of programs, then, is greater now than ever. 
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Specific Challenges to Evaluation in Tennessee: Results of Survey Analysis 

With the threat of continued reductions in funding of McKinney programs for homeless 
children and youth, the imperative to demonstrate the success of programs grows stronger 
each year. Yet specific challenges remain for programs in Tennessee that make local 
evaluation of progress difficult: 

Commitment to Evaluation for Accountability. The clear link between Congressional 
funding levels for McKinney programs and political pressure at the national level to cut 
social programs has led many state and local coordinators to be defensive about the 
programs they currently offer. Most coordinators use the strategy of documentation of 
services to defend why their program is worth preserving. Evaluation is rarely used at the 
local level as a tool for continuous improvement or accountability to stakeholders for 
specific educational outcomes. 

Uncertain Length of Stay in Programs. While the law permits homeless education 
programs to serve students for the entire school year in which they become homeless, 
even if they move to a more permanent residence, often students are involved in such 
programs for a short period of time. In Nashville, for example, the average stay in the 
program was six weeks (Penuel, Davey, & Allison, 1996). Programs may provide 
tutoring and other social services to students twice or three times a week during that 
period, but if students leave the shelter and/or school, continuing to monitor the progress 
of those students becomes difficult. Only rarely can homeless education program 




IB 



EVALUATION OF HOMELESS EDUCATION 

16 

managers monitor this process so that students are not lost to the program. Moreover, if 
the program serves students on site at a shelter, providing services to students beyond the 
shelter stay is dependent on the availability of transportation for students and on the 
willingness of parents to continue supporting their child through tutoring at the program. 

The short length of stay poses a difficulty for choosing a unit of analysis for 
educational and social progress. In some cases, the student may be in the program for 
less than a six weeks grading period, while in other cases, the student may remain in the 
program for three to four months and have assessment data that can demonstrate whether 
the student has made significant progress or not. Often, there is time for a social- 
psychological intake, but students leave the shelter before a social worker or psychologist 
can follow-up and assess a student’s progress. The unit of analysis for assessment, then, 
has to be short enough to match the average length of stay for a student in the program 
and yet long enough to show measurable progress in academic and/or social 
development. 

Size of Ns for Academic Programs. A related problem of measuring the effects of 
programs is the number of students served in smaller programs. In an urban district like 
Nashville or Memphis, a program can serve intensively between 200 and 500 students, 
but smaller districts may serve under 50 students, and many of these students may have 
been in the program for a short duration of time. Even within larger districts, the number 
of students who may have received intensive services— such as tutoring, transportation 
assistance, social work services— may include a much smaller proportion of students. 




17 



EVALUATION OF HOMELESS EDUCATION 

17 

Knowing how these students differ from the wider group of students served on anything 
other than demographic variables may be difficult for programs to ascertain. 

Developing a single assessment tool that is useful for all the students in a program 
is similarly challenging. Most programs have students from a full age range: Table 2 
shows the distribution of students by grade for Memphis’ homeless education program. 



Table 2. Grade Distribution of Homeless Education 
Program Students in Memphis 



■Grade 


f Number and 
Percentage 1 of 
Students 


K 


31 (13%) 


1 


35 (15%) 


2 


28 (12%) 


3 


19(8%) 


4 


28 (12%) 


5 


24(10%) 


6 


16(7%) 


7 


9 (4%) 


8 


1 1 (5%) 


9 


15(7%) 


10 


6 (3%) 


11 


0 (0%) 


12 


1 (<1%) 



While many standardized tests have different, but comparable versions for students in 
different grade levels, the forms used by the state are not available to teachers. The 
number of students in each grade, moreover, makes statistical comparisons difficult for 
the middle grades and high school students. 



Accuracy of School Records School attendance records in many districts in Tennessee 
are most accurately kept in the cumulative file of students. Homeless education 
coordinators and program evaluators in larger districts, however, must often rely on 
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electronic data systems at a central office level to track students who have left the shelter 
and possibly transferred to another school in the district: students they track may attend 
between 30 to 40 different schools in the district. The students they are tracking, 
however, may not be in this centralized database of attendance information. The schools 
may be slow in transferring records, or there may be two records for a student, one 
showing the student has missed a large number of days of school with another indicating 
they had been in attendance at another school during the days marked absent by the first 
school. Attendance records from the Nashville program in 1995-96 show how 
problematic these records are for calculating school attendance: 

Table 5. Nashville HERO Homework Students in Attendance Tracking System 

(source: Penuel, Davey, & Allison, 1996) 



According to their annual evaluation report for 1995-96, in Memphis some 8% of all 
students in the database either never registered for school officially or their whereabouts 
were unknown. 

Diversity and Location of Services. Tennessee’s homeless education programs each 
have a different core focus. In two large cities, the focus is on providing transportation to 
homeless children and youth to ensure they can go to their “school of origin,” that is, the 
school they were attending before they came homeless. In two other large cities, the 



Status of Records 



# of students 



Accurate attendance records 

Do not appear in school system database 

"Lost," whereabouts unknown 



105 

38 

9 
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focus is on academic tutorial services. In smaller districts, funds are evenly distributed 
among tutorial, social development, and transportation activities. The diversity of 
services and distribution of resources makes it difficult to assess powerfully the impact of 
one set of services for homeless children and youth. Tutorial services, moreover, are not 
the same across all sites. Some employ certificated teachers,, while others rely on 
instructional support staff. Some provide homework assistance, while others also provide 
direct instruction to students. 

The location of services poses additional challenges to program evaluation. Many 
programs in Tennessee offer tutorial and other services at the shelter site. The tutors may 
have little direct contact with other service providers in the schools, unless that contact is 
mediated by the program coordinator. School staff may have critical information about 
students that is lost when the shelter staff, families, and the direct experience of tutors 
and social workers are the only source of information about a students’ educational and 
social needs and progress. 

Measurement of Social Development Outcomes. Most programs reported on their 
surveys that they believed their programs built students’ self esteem; yet none of the 
programs currently used a measure to gather baseline and evaluative information about 
social development effects of their programs. While large scale research studies have 
been able to measure these outcomes (Bassuk & Rubin, 1987; Davey, 1995), smaller 
scale programs often have difficulty identifying appropriate measures of social 
development and finding staff skilled in analyzing their results. As a result, key 
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outcomes of homeless education programs may go unmeasured, despite the fact that these 
outcomes may be central to the efforts of McKinney Act programs. 

Strategies for Addressing the Problems of Evaluation: The Case of Nashville 

Evaluation for Accountability and Program Improvement. As part of our work with the 
Nashville program, we facilitated the development of program objectives and activities 
for the current grant cycle. In this facilitation process, we asked local school officials to 
invite various stakeholders to a series of meetings to plan the program. The program 
planning process invited participants to: 

• define a set of “distant objectives” for their program; that is, objectives 
that could be obtained over a longer period of time with students; 

• define “nearby objectives” that could be met after one or two sessions 
with a student (e.g. completion of homework); 

• define a set of activities that would help reach the nearby and distant 
objectives; and 

• articulate the links between activities and objectives. 

During this process, we helped participants identify a set of indicators for each nearby 
and distant outcome they named. These nearby indicators, as we called them, could 
serve as an important role in the evaluation process: even if long-term or distant outcome 
data were unavailable, the program could regularly track each time a student got to 
school safely and on time; each time a student completed a homework assignment during 
a tutoring session; or each time a student received clothes to wear to school. 
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What resulted from this planning process was a clearer grant application than had 
been submitted in the past and a well-defined evaluation plan. The new coordinator in 
Nashville, who participated in the process, reports that grant now serves as a tool to guide 
her data collection activities. She feels that the process of collecting data on nearby 
indicators makes the evaluation process more manageable and tangible than it would be if 
the evaluation focused only on distant outcomes such as student achievement and self 
esteem. It frames the relationship with the evaluator (the second author) for the program, 
and gives a clear action plan for completing the evaluation. 

Length of Stay/Size of Ns. This remains a challenging problem for the program 
coordinator. In particular, she has found that there is no set pattern of services that she 
can offer to all families to measure program effectiveness. She finds that some families 
require much individual attention (transportation, clothing, et cetera), while others are 
fairly independent, using only the tutoring program at the shelters. A small percentage of 
families occupy about half of the coordinator’s time. 

In the face of these challenges, the coordinator has developed specific forms that 
can be used by each tutor and by her to document services provided, however diverse 
they might be. She has also revised an earlier version of a Teacher Report Form, 
developed by the first author (Penuel), that invites tutors to rate each student weekly on 
behavior and attitude (see attached form). The form asks questions about how well the 
student listens, works cooperatively, shows self controls, and successfully completes 
homework. The form also asks about students’ attitudes toward and engagement with 
learning tasks and school. In the past, student engagement with learning has been 
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identified by teachers in Nashville as a key positive outcome of the program (Penuel, 
Davey, & Allison, 1996). 

Accuracy of School Records. There has been no way to fix the two to three week delay in 
getting accurate attendance records for students when they change schools. The program 
this year has hired a quarter-time attendance worker, facilitated by the fact that the 
program is sponsored by the Social Work and Attendance Division in the public schools 
in Nashville. Thus far this year, the coordinator has found that there is no substitute for 
the time-intensive process of calling schools directly to find out the truancy status of 
particular students who miss a lot of school. Experience in other districts has been 
similar (Cronin, personal communication). Even where electronically-uploaded data 
exists, accurate data can only be achieved when persons integrate and evaluate various 
sources of data for reliability and accuracy. As a policy matter, districts seeking to serve 
homeless children and youth must allocate more resources to people whose job it is to 
make sure data is up-to-date and reliable. 

Diversity and Location of Services. The original grant application for this year called for 
a full-time social worker to assist with linkages between the program and the schools 
where students attend. Providing services at the shelter has been in the past the most 
effective site of service delivery, since the number of shelters in Nashville is small and 
since there is no risk of singling out homeless students at their schools. At the same time, 
the large number of schools attended by students and limited program budget has meant 
that there has been little regular contact with classroom teachers. The position outlined in 
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the grant application called for regular contact with the classroom teacher to identify 
specific learning needs for tutored students. In this way, a better bridge could be built 
between the program and individual schools not only for more targeted services but also 
for better and more complete information on student progress. 

Unfortunately, this position was not funded by the state. To date, the coordinator 
reports this aspect of program evaluation remains a challenge. The challenge is made 
even more difficult by the continued inability to identify more students who are living 
doubled-up in public housing who could benefit from the program. Serving such youth 
would require more than just identifying those students (a problem in itself) but would 
also require the program to provide transportation to tutoring services currently being 
offered at shelters. Whether students and families would be interested in such services, 
even if they could be provided, would be difficult to ascertain. 

Measurement of Social Development Outcomes. Our research team (Penuel, Davey, & 
Allison, 1996) has been developing interview protocols over the past two years to help 
assess social development outcomes of homeless education programs. Among the topics 
covered in these protocols are: student perceptions of the shelter as a home (see Penuel & 
Davey, 1998); friendship development; and experiences in school. We interviewed some 
fifty students using the protocol in 1996-97, and administered the Teacher Report Form 
of the Child Behavior Checklist (Achenbach & Edelbrock, 1986) and the Harter Self- 
Concept Profile (1985). This year, the coordinator, who is a trained social worker, is 
administering both the Coopersmith Self-Esteem Inventory and Harter Self-Concept scale 
to students in the program, as a way of collecting social development information on 
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students in the program. The Harter in particular is a useful scale, as there are different 
versions of the scale available for kindergarten through early adolescence. 



Conclusion: Implications for Design of McKinney Program Evaluations 



As the problems in Tennessee highlight, the challenges of designing and 
implementing evaluations of homeless education programs are numerous, and not all of 
them are readily solved. We have worked closely with programs across the state, but 
particularly in Nashville, to develop systems for addressing some of those problems, and 
we believe it is important for researchers to develop such a collaborative role with 
programs. The aim of our collaboration is not simply to conduct the evaluation but to 
work closely with staff who work with students directly to define what kinds of process 
and outcome data are useful to collect and what systems would need to be in place to 
support evaluation of their homeless education program. This process, even when it does 
not find solutions to every problem, has been helpful in Nashville, where each year the 
evaluation process has improved in its ability to track student process and outcomes. 

Many of the solutions for conducting local program evaluations do not lie with 
the programs themselves but within the district, the state, and national levels. It is 
imperative that researchers support program staff in understanding which solutions can 
be constructed at each of those levels. In Nashville, for example, the program evaluation 
would benefit from the assignment of staff in the district who could check attendance 
data for accuracy and make sure such data are up-to-date for each student. The state, 
meanwhile, might assign an evaluation assistant to work closely with programs each year 
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as part of its support system. Such an evaluation assistant might play the role that we the 
authors have been able to provide in Nashville, helping to design strategies for serving 
homeless children and youth and assisting with evaluation. At the national level, more 
stable funding and clearer direction on program objectives would enable states and local 
programs to plan more effectively and design evaluations that could effectively defend 
the program against critics. In the end, until each of these levels of educational 
governance coordinates the evaluation work, McKinney programs will remain programs- 
at-risk, just as the students they serve remain in great need of educational opportunity. 
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